A Cost-Sensitive Visual Question-Answer Framework for Mining a Deep And-OR Object Semantics from Web Images

This paper presents a cost-sensitive Question-Answering (QA) framework for learning a nine-layer And-Or graph (AoG) from web images, which explicitly represents object categories, poses, parts, and detailed structures within the parts in a compositional hierarchy. The QA framework is designed to minimize an overall risk, which trades off the loss and query costs. The loss is defined for nodes in all layers of the AoG, including the generative loss (measuring the likelihood for the images) and the discriminative loss (measuring the fitness to human answers). The cost comprises both human labor of answering questions and the computational cost of model learning. The cost-sensitive QA framework iteratively selects different storylines of questions to update different nodes in the AoG. Experiments showed that our method required much less human supervision (e.g. labeling parts on 3–10 training objects for each category) and achieved better performance than baseline methods.